Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: sids Date: Mar 14, 2012 9:12pm
Forum: web Subject: Page cannot be crawled due to robots.txt

This site used to work perfectly www.usmlein.com

Now get this robots message. Please could it be fixed please

thanks again

Reply to this post
Reply [edit]

Poster: Jeff Kaplan Date: Mar 14, 2012 10:44pm
Forum: web Subject: Re: Page cannot be crawled due to robots.txt

the current owner of the domain has a robot.txt file disallowing crawling. the pages cannot be displayed unless it is removed.

http://www.usmlein.com/robots.txt

Reply to this post
Reply [edit]

Poster: sids Date: Mar 15, 2012 8:49am
Forum: web Subject: Re: Page cannot be crawled due to robots.txt

dont understand it was visible before last week why changed now really needed to see its contents tbh

anyway to get round this